Visual Imagination from Texts

Imagination is a fundamental ability of humans which resides in the cognitive system. We propose a connectionist model that generates images from a given sentence after trained on a dataset of image-sentence pairs. The model is composed of language model and image model that are connected with a latent variable constrained by a prior distribution. The latent variable encodes dual information and it is generalized by Bayesian learning method. We trained on cartoon video series ‘Pororo’ and 16,066 fine-grained sentences describing short clips. Our model successfully generates plausible images which are highly correlated with a given sentence.